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Achievement  of  Students  from  Groups  Inst  rue  trrl 
by  Pro^rainetJ   M.Ueri.ils,    (31.1  ss  room    I'ck  h«"r,    or    Both 

O.    Robert  Brown,    Jr. 

The  purpose  of  the   study   is  to  investij>ate  effects  of  different  strategies  for 
extended  classroom  use  of  some   UICSM  programed  texts;  students   spent  from 
eight  to  ten  weeks  studying  our  materials.      This   report  summarizes   some  of  the 
data  gathered  during  the  first  semester  of  the    196Z-63    school  year.  ^ 

The  data  consist  of  results  on  achievement  tests  based  on  the  content  of  both 
the  sequence  of  programed  texts  prepared  by  the  UICSM  Programed  Instruction 
Project  and  also  UICSM  Unit    1,    a  textbook  on  introductory  algebra.      The  experi- 
mental samples  were  chosen  from  ninth  graders  in  twelve  classes  in  eight 
schools,    the  control  samples  from  ninth  graders  in  eight  classes  in  eight  schools. 
[See  Appendix   A.  ] 

In  one  experimental  treatment  ["pure"   condition]  the  programed  materials 
were  the  sole  agents  of  instruction,    used  in  place  of  the  classroom  teachers.      The 
other  experimental  treatment  ["anticipating"   condition]  involved  a  UICSM-trained 
teacher  using  the  programed  materials  to  precede  the  usual  classroom 
development  of  topics.  2     The  control  group  students  were  instructed  by  a 


iSee  Semi-annual  Report  (Quarterly  Reports    7    and    8)  for  December  6,    196  2  - 
June    6,    1963. 

2A  third  experimental  treatment  ["following"  condition]  was  devised  such  that 
the  programed  materials  were  used  after  the  classroom  teacher's  development 
of  topics,    but  inadequate  data  are  available  for  valid  conclusions.      Appendix    C 
shows  the  bimodal  nature  of  the  ability  [and  achievement]  distributions  for  the 
students  involved. 


UICSM-trained  teacher  using  the  regular  UICSM  Unit    1;  no  programed  materials 
were  used  in  the  control  classes. 

METHOD 
Pedagogy 

The  UICSM  has  been  in  the  forefront  of  the  movement  toward  a  modern 
secondary  school  mathematics  curriculum.  ^    Since  all  the  materials  —  experi- 
mental programed  text  and  regular  text  —  and  all  the  teachers   —  "progranned" 
classes  and  regular  classes   —  referred  to  in  this  study  were  UICSM-produced 
or  UICSM-trained,    we  must  consider  characteristics  of  the  UICSM  in  order  to 
interpret  the  findings.     UICSM  materials  are  carefully  designed  to  promote,    and 
often  require,    extensive  use  of  discovery  techniques  in  teaching.     UICSM  teachers 
are  trained  in  leading  students  to  develop  their  own  problem- solving  procedures 
and  short  cuts. 

In  addition  to  using  discovery  techniques,    UICSM  strives  for  internal  con- 
sistency and  cohesiveness,    and  for  careful  use  of  terminology.     It  considers  the 
teaching  process  as  a  continuing  dialogue  with  a  minimum  of  "  tell-and-do" ;    the 


®For  a  brief  description  of  the  philosophical  orientation  of  the  UICSM,    see 
Max  Beberman.     An  Emerging  Program  of  Secondary  School  Mathematics. 
Inglis  Lecture  Series.      [Cambridge:     Harvard  University  Press,    1958.] 


studfiil   IS    Ifcl  to  w.iiU   ti)  (Id   sonu'tlun^  aiul   llun   Id  try   to  li^iirt-   out  lor   hiiriHi^lf 
how  to  do  it.      Wo   have   nuiclr  the  sainc  emphases   in  writing  the  programed  texts. 

Bear  in  mind  that  our  programed  texts  are  designed  to  approximatt;  what  a 
student  encounters  in  a  carefully  conducted    UICSM    class.      The  programs, 
insofar  as  possible,    do  the  same  things  that  we  have  encouraged  teachers  attend- 
ing our  training  institutes  or  watching  our  demonstration  classes  to  do.'^     All 
mathematical  concepts  treated  are  foreshadowed,    explored,    introduced,    and 
developed  within  the  programed  texts  themselves  (UICSM  Staff,    1963). 

We  can  characterize  our  study  as  an  investigation  of  three  ways  of  imple- 
menting ini>truction  within  one  specific  framework  for  teaching.      The  implemen- 
tations are: 


(i)     use  only  programed  instruction 
(ii)     use  only  teacher  instruction 
(iii)     use  complennentary  programed  and  teacher  instruction, 


and  the  framework  is: 


have  each  student  himself  "discover"  almost  all  the 
operational  and  organizational  "  rules"  for  the  mathe- 
matics he  is  in  the  process  of  learning. 


4  An  interesting  by-product  of  our  approach  to  programing  is  that  the  programed 
texts  we  produce  can  be  used  for  training  teachers  in  the  presentation  of  UICSM 
curriculum  materials.     The  basic  pedagogical  principles  we  follow  can  be 
pointed  out  very  explicitly  as  the  development  of  topics  in  the  programed  texts 
vinf  olds . 

We  have  already  used  a  revised  edition  of  our  programed  materials  to  assist 
two  teachers  who  had  to  teach  UICSM  Unit    1    without  having  had  the  training  we 


Materials 

Programed  Instructional  Parts.     The  programed  instructional  parts 
[plastic -bound  booklets],    were  prepared  by  UICSM    staff  members .     The 
programers  were  competent   UICSM    classroom  teachers  who  had  gained  expe- 
rience in  programing  during  our    19^1-62    study.     They  used  both  linear  and 
branching  techniques  and  a  variety  of  formats  and  styles.      The  books  contain 
numerous  illustrations  and  occasional  discussions  of  previous  problems  or  new 
ideas.     The  treatnnent  of  each  new  topic  is  aimed  at  an  audience  naive  to  that 
topic,    giving  the  books  a  developmental  emphasis.     The  regular  text,    which 
served  as  the  basis  for  the  sequence  of  parts,    is  Unit    1    of  HIGH  SCHOOL 
MATHEMATICS.  5     Appendix    B   lists  the  topics  covered  in  each  of  the  programed 
parts. 

Ability  and  Aptitude  Tests .     Each  student  took  as  pretests  the  Test  of 
General  Ability    (SRA)    and  a  mathematics  form  of  the  Sequential  Tests  of 
Educational  Progress    (ETS),    referred  to  as    TOGA  and   STEP,    respectively. 


have  found  is  usually  necessary.     Each  had  two  classes,    let  the  programed  texts 
be  sole  agents  of  instruction  in  one  class,    and  used  them  with  Unit    1    as  desired 
in  the  other  class.     The  students  did  acceptable  work  on  the  final  examination  and 
the  teachers  reported  that  they  had  "learned  a  lot*'. 

5Max  Beberman  and  H.   E.    Vaughan.     HIGH  SCHOOL  MATHEMATICS,  Unit    1. 
[Urbana:    University  of  Illinois  Press,    I960]. 


T.ibli-    1 
AchicvtMm'nt  tosts   for  usi-  witli    196Z-()i    pro^  r.iiiicd  trillion  of    UKJSM    Unit    1 


Il'St 


Programed  Paris   Covered 


Text  Pages   Covered 


Unit    1 
Exam 


101 

lOZ-103 

104 

105 

106-107 

108,   109,   110,   110.5 

111,   HZ,   113,   114, 
114.5 

115 
116 
101  through    116 


1-Aj  through  [l-Oj 
1-1] through  [1-15J 
1-16]  through  [1-23] 
1-24]  through  [1-33] 
1-33]  through  [1-42] 
1-43]  through [1-62] 
1-63]  through  [1-95] 
1-95]  through  [1-102] 
1-103]  through  [1-110] 
all  of  Unit    1 


*  Appendix   B   gives  the  number  of  pages  in  each  part  and  the  topics  covered. 
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Scores  on  these  two  tests  were  used  in  identifying  the  students  who  made  up  the 
matched  pairs  used  in  our  analyses. 

Achievement  tests.     Each  achievement  test  was  constructed  to  cover  . 

explicitly  the  material  treated  in  the  related  programed  parts.      Table    1    summa- 
rizes the  tests  and  related  part(s). 


Tests  were  administered  as  soon  as  possible  after  a  student  in  an  experi- 
mental group  had  completed  the  necessary  part(s)  or  as  soon  as  a  control  class 
had  covered,    to  its  teacher's  satisfaction,    the  requisite  content.     A  standard 
summary  examination,    the  UICSM  Unit    1    Examination,  ®    was  also  used. 
Conditions 

Experimental  Treatments,    (P)  and  (A).     Here  are  descriptions  of  the  two 
experimental  treatments  reported  on  in  this  paper: 

"Pure"     (P) 

The  only  instruction  students  receive,    except  for  unusual 
circumstances,    is  by  means  of  the  programed  texts. 
Students  work  in  the  materials  throughout  each  class 
meeting,    except  when  taking  appropriate  achievement 
tests.     Homework  assignments,    when  given,    consist  of 
additional  work  in  the  programed  text. 


6UICSM  HIGH  SCHOOL  MATHEMATICS,    Unit    1 .     Examination  [Urbana: 
University  of  Illinois  Press,    I960]. 


"  Anlicip.itin^"     (A) 

Tlu*   studiMits    receive   instruction  botli  I  roin  the  pronramecJ 
texts  and  from  their  regular  classroom  teacher. 
Assigruiients  either  for  homework  or  for  in-class  work 
are  given  in  such  a  way  that  every  topic  the  teacher  dis- 
cusses has  been  anticipated  by  its  treatment  in  a  programed 
text.      As  these  texts  give  the  introduction  of  topics,    the 
teacher's  discussion  includes  elaboration  and  clarification 
of  new  topics.      In  addition,    the  teacher  is  encouraged  to 
give  a  different  perspective  on  the  topic  and  help  clear  up 
general  or  individual  difficulties  of  the   students. 

The  names  of  the  treatments  were  chosen  to  show  the  order  and  extent  of 

interaction  between  the  programed  texts  and  the  teacher. 

Control  Condition  (C).      The  eight  control  class  teachers  were  aware  of  the 
fact  that  students  from  their  classes  were  being  used  as  controls  in  a  study 
evaluating  uses  of  programed  materials.     It  may  be  because  of  this  that  these 
teachers  took  more  time   [and  care?]  to  complete  their  teaching  of  Unit    1    than 
comparable  teachers  in  earlier  years.      The  control  teachers  were  requested  to 
use  only  the  usual  UICSM  teaching  techniques  and  materials. 

Treatment  Pairs.      The  three  implementations  mentioned  earlier  give  rise 

to  these  three  questions: 

(1)     Do  the  programed  texts  as  sole  agents  of  instruction 
[condition    P]  teach  as  effectively  as  a  UICSM-trained 
control  teacher? 


(2)     Do  UICSM-trained  teachers  using  the  programed  texts 

[condition  A]  teach  more  effectively  than  UICSM-trained 
control  teachers? 


(3)      Do  UICSM-trained  teachers  using  the  programed  texts 
teach  more  effectively  than  the  programed  texts  as  sole 
agents  of  instruction  [condition    P]  ? 

Results  under  the  "pure"  mode  relate  to  the  first  and  third  questions;  those  from 

the  "anticipating"  mode  relate  to  the  second  and  third  questions.      The  control 

classes  are  used  for  questions    1    and    2. 

Of  course,    these  three  questions  deal  only  with  the  implementation  of  the 
finished  programed  texts.     Basic  to  the  actual  preparation  of  the  materials  Avere 
questions  dealing  with  such  matters  as  step  size  and  sequence,   implementation 
of  a  discovery  method  of  teaching,    effective  use  of  branching,    etc. 
Students 

Pure  vs.    Control  (Question    1).      There  were  five    P   classes  and  five    C 
classes  from  schools  whose  students  came  from  very  similar  socioeconomic 
environments  and  which  had  very  similar  classroom  conditions.     We  selected  all 
students  from  these  ten  classes  who  had  a  reasonably  cornplete  record  [at  least 
scores  on  the  two  pretests  and  achievement  tests    1-3,    5-9]  and  constructed 
matched  pairs  of  students  (P,    C)  such  that  members  of  each  pair  had  identical 
STEP  (scaled)  scores  and  TOGA   IQ  estimates  within   5    IQ  points  of  one  another. 


Table    I 

Means,  SD's  aiid  \_  ratios   lor   C-ontrol(CO  and    Pure(P)  Matched    Pairs 

Means  SD's 

Hi  £  £  c  p  t_ 

STEP          90  Z82.Z1  Z82.  21  8.35  8.35 

TOGA        90  129.56  129.56  10.59  10.30 

1  90  18.70  18.56  3.00  3.74  .680 

2  90  22.97  21.39  4.02  5.54  2.236v/ 

3  90  19.23  19.04  4.15  4.37  .813 

4  72  12.40  9.86  1.87  2.66  6.69    v/ 

5  90  9.52  6.50  3.79  3.23  7.907v/ 

6  90  23.60  21.34  5.08  4.64  3.748/ 

7  90  19.40  17.66  3.63  4.66  4.091v/ 

8  90  17.22  16.74  2.78  3.21  1.333 

9  90  8.64  7.36  1.92  2.79  4.079^/ 
Total          72  156.00  141.04  19.72  25.81  4.908/ 


number  of  inatched  pairs 
/significant  for   a   =    .05 
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Table    3 
Means,    SD' s  and   t_  ratios  for  Anticipating  (A)  and  Control(C)  Matched  Pairs 

Means  SD' s 

Nl  A  C  AC  t_ 

STEP        53         281.02  281.02  6.63  6.63 


3GA 

53 

125.55 

125.77 

12.28 

12.  24 

-~- 

1 

53 

19.83 

18.40 

3.39 

2.85 

2.455     y 

2 

53 

22.51 

22.79 

4.79 

3.95 

-.402 

3 

53 

18.38 

18.60 

4.00 

4.45 

-.370 

4 

40 

10.60 

12.10 

3.23 

2.58 

-2.732v/ 

5  53  9.53  8.87  3.92  3.81  .918 

6  53  22.91  24.15  4.69  4.69  -1.592 

7  53  18.94  19.06  4.62  3.63  .107 

8  41  17.80  16.60  1.98  2.46  1.377 

9  41  8.92  8.80  2.01  2.53  .314 
Total  33  153.91  157.39  23.14  21.77  -.629 


^number  of  matched  pairs 
\/significant  at  a   =    .05 
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I'.iblt-    4 
Moans,    SD's  and  J^   ratios   lor  Antic  ipatin^(  A)  and    Pure(P)  Matched   Pairs 

Moans  SD's 


PA  P 


STEP  SZ  Z83.ZZ  ZSi.ii  7.38  7.38 

TOGA  51  130.25  130.21  10.16  10.28 

1  52  19.19  20.37  3.51  3.35  1.992 

2  48  22.08  22.54  5.27  5.41  .644 

3  52  19.92  19.37  3.62  4.33  -.799 

4  52  10.52  10.81  2.66  3.26  3.789^/ 

5  52  7.15  10.00  4.04  4.24  4.031^ 

6  52  22.67  22.98  4.71  5.04  .037 

7  52  19.17  19.75  4.57  4.54  .729 

8  43  17.70  18.28  2.21  1.67  1.704 

9  43  8.40  8.91  2.37  2.04  1.512 
Total  40  149.40  157.33  23.70  20.04  1.616 


number  of  matched  pairs 
^significant  at   a   =    .05 
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Table    5 

Means,    SD's  and  Percentile  Range  on  UICSM  Unit    1    Examination 
for  the  Norming  and  the  Sample  Populations 


Group 

N 

Mean 

SD 

%-ile  range 

Norming 
Population 

2000 

16.2 

4.9 

40-47 

Sample  Populations 

Ci 

62 

18.6 

3.  7 

54-62 

Pi 

62 

16.7 

4.4 

40-47 

As 

34 

18.3 

3.7 

54-62 

C^ 

35 

18.7 

4.0 

54-62 

A3 

47 

18.7 

3.7 

54-62 

^3 

46 

17.4 

4.0 

47-54 

13 

A  total  of    ^H")    pairs  was  ob)tainrd   in  this  way.       Tahlr    I    lists   the   tiic.ins  .irifi    SD's 
on  the  pretests  and  on  our  specially  constructed  achievement  tests  for  the    P 
and    C   groups  finally  determined. 

Anticipating  vs.    Control     (Question    2).      There  were  three  classes  which 
participated  under  condition    A.      We  selected  all  students  who  had  a  reasonably 
complete  record  [defined  above]  and  used  as  our    C    students  exactly  those  found 
in  the  selection  from  the  five    C   classes  used  above.     Matched  pairs  of  students 
(A,    C)  were  formed  just  as  in  the    P  vs.     C    situation.     A  total  of    53    pairs 
resulted,    and  Table    3    lists  the  related  means  and    SD's. 

Anticipating  vs.    Pure     (Question    3).     All    A    students  eligible  for  pairing  in 
A  vs.     C   and  all    P   students  eligible  for  pairing  in    P  vs.     C   were  considered  in 
the  construction  of    52   matched  pairs  of  students  (A,    P)  vv'ith  reasonably  complete 
data.     Table    4   lists  the  means  and    SD's  of  the    A  and    P  groups  so  determined. 


Note  that  we  have  used  what  amounts  to  a  sampling -with- replacement 
scheme  in  the  construction  of  our  matched  pairs.     It  is  possible  but  not  necessarily 
the  case  that  a    P   student  could  be  in  both  the    P  vs.     C   and   A  vs.     P   samples. 
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A  similar  remark  holds  for  the    A    students  and  for  the    C    students.      Whenever 
we  need  to  differentiate  between  the  two    P    samples  used  for  questions    1    and    3, 
we  write  'P-^'  and  'P3'.     Similarly,    we  write  '  C^^'  and  'Cg'  to  differentiate 
between  the  two    C    samples  for  questions    1    and    2,    'A^'  and  'Ag'  to  differentiate 
between  the  two    A    samples  for  questions    2    and    3, 

Table    5   lists  the  means  and    SD' s    on  the  Unit    1    summary  examination  of 
the  various  groups  described  above. 

RESULTS 
The  Tests  Individually  J 

Pure  vs.    Control.      The  data  presented  in  Table    2    lead  us  to  conclude  that 

our  programed  version  of  Unit    1    used  as  the  sole  agent  of  instruction  failed  to 

teach  as  effectively  as  control  teachers.     Six  of  the  differences  on  achievement 

tests  were  significant;  in  each  case  the  direction  of  the  difference  favored  the 

control  group.     Consequently,    the  question: 

(1)  Do  the  programed  texts  as  sole  agents  of  instruction 
[condition  P]  teach  as  effectively  as  UICSM-trained 
control  teachers? 

must  be  answered  'No'. 

It  is  interesting  to  note,    however,    that  on  the  standardized  Unit    1 

Examination  [see  Table    5]  the  mean  of  the    C   group  clearly  exceeded  that  reported 
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\ov   I  lu"   luirnuiif.;   f)0()ul<it  n>ii  '     .iiid  fvrii  tin-   mraii  of    t\u-     P    ^rou|j  wdK   sli^hlly 
urt'attT,    bill  not   si^nilicantly  so.       This    rt'sult  U-ads  to  the  conji-cturt-  that  the 
greater  length  of  time  spent  on  the  Unit    1    materials  by  the    C    group  tended  to 
make  their  performance  better  than  would  have  been  predicted.      Students  in  the 
P   group  probably  did  not  spent  as  much  time  as  the    C    students  since  no  home- 
work assignments  were  given  during  the  first  six  weeks  of  instruction. 

Anticipating  vs.    Control.     The  data  presented  in  Table    3    lead  us  to  conclude 
that  teachers  using  our  programed  texts  to  treat  topics  before  giving  the  topics 
any  classroom  discussion  were  just  as  effective  as  control  teachers.     Two  of  the 
reported  differences  are  significant  but  the  direction  of  the  difference  favored  the 
A   group  once  and  the    C   group  once.     Looking  at  the  Unit    1    Examination  scores 
we  see  that  the    A   and   C   groups  had  comparable  means  which  were  above  that  of 
the  norming  population. 

On  the  basis  of  the  Table    3    comparisons,    the  question: 

(2)     Do  UICSM-trained  teachers  using  the  programed  texts 

[condition   A]  teach  more  effectively  than  the  UICSM-trained 
control  teachers  ? 

must  be  answ^ered  'No'.     We  can  say,    however,    that  our  teacher -prog  ram 


"7 The  populations  used  in  this  study  are  comparable  to  the  norming  population  for 
the  Unit    1    Examination  in  that  our  students  come  from  the  upper    Z/3    of  the 
STEP  distribution  and  the  students  in  the  norming  population  came  from  the 
upper    2/3    of  the  DAT  Numerical  Ability  test  distribution. 
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combination  was  certainly  no  less  effective  than  control  teachers.     In  addition, 

comments  from  teachers  revealed  that  working  under  condition   A  had  extra 

benefits,    such  as  increased  knowledge  of  individual  student's  problems. 

Anticipating  vs.    Pure.      The  data  presented  in  Table    4    show  two  significant 

differences  between  the    A  and    P  groups,    both  in  favor  of  condition  A.     Since  it 

is  also  the  case  that  eight  of  the  nine    A  means  were  numerically  greater  than 

the  corresponding    P  means,    we  can  answer  the  question: 

(3)     Do  skilled  UICSM  teachers  using  the  programed  texts 

[condition   A]  teach  more  effectively  than  the  programed 
texts  as  sole  agents  of  instruction  [condition    P]  ? 

with  a  '  Yes' . 

Finally,   we  note  as  a  purely  descriptive  result  that  the    P^    and    Pg   means 

and    SD's    from  Tables    2    and   4   gives  us  an  estimated  minimum  achievement  leve! 

for  students  using  the  programs  without  any  teacher  assistance.      For  example, 

a  student  who  studied  Part    101    independently  could  be  expected  to  get  a  score  of 

18    or    19   on  the  related  achievement  test.     Such  predictions  must  be  evaluated  in 

the  light  of  the  circumstances  for  the  individual  students,    but  the  ability  to 

predict  a  certain  level  of  achievement  for  a  given  student  on  a  given  topic  is  usefu 


to  classroom  teachers. 
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Figure  1.     Cvimulative  mean  scores  on  achievement  tests. 
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Table    6 

Sign-test  summary  by  achievement  test  and  paired  groups 

Groups 
Number 

Test        of  items                  C-P                                     A-C  A-P 

Nb'+'         Nb'-'                Nb'+'         Nb  ' -'  Nb  ' +'  Nb  ' -' 

1  31                   14       ■           16                         18               8  20  8 

2  30                  21                     7     v/                  13            16  15  10 

3  30                   17                   12                         15            12  10  16 

4  15                   15                     0     y                   1             13     •^  9  5 

5  17                   15                     2     v/                 11               6  13  2     v/ 

6  37                   30                     6     v/                 10             19  24  12 

7  30                  23                    7     y                   9            15  16  10 

8  20                   li                     8                         12              7  11  4 

9  11                     9                     2     y                   45  73 


v/ 

Significant  at   a   =    0.  05 
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The  Tests  Cumulatively 

Another  way  to  look  at  performance  on  the  achievement  tests  both  within 
groups  ol  pairs  and  between  groups  of  pairs  is  to  consider  cumulative  mean 


scores.      Figure    1    gives  a  graphical  presentation  of  continuing  progress  through 
the  material  for  each  condition  in  each  group  of  pairs.     Also  given  is  the  maximvun 
possible  cumulative  score.     In  a  sense,    each  treatment  group  is  treated  as  an 
individual  whose  nine  achievement  test  scores  are  the  respective  group  means  for 
the  tests.      Each  "individual"  is  identified  by  a  letter  corresponding  to  the 
treatment  he  received  subscripted  by  a  numeral  corresponding  to  the  question 
for  which  his  data  were  used. 

Looking  at  the  first  pair  of  curves  we  see  that  the  "individuals"    C^^    and    P^ 
become  slowly  but  surely  dissimilar,    C^^    gaining  over    P^.      The  graph  gives  us 
an  indication  of  the  nearly  constant  rate  of  gain.      Considering    Ag   and   Cg   we  see 
that  the  overall  effects  are  almost  identical.      The  cumulative  effect  on   Ag    and    Pg 
is  more  like  that  on   C^   and    P^,    but  clearly  not  as  pronounced. 

Looking  across  the  graphs  we  see  that  the  final  positions  of  individuals  with 
the  ScLme  treatment  but  in  different  pairs  do  not  agree.      [See  also  Appendix    D.  ] 


zo 

It  would  be  useful  to  have   results  in  which  three  students  matched  for  ability, 
one  from  each  condition,    could  be   related.      A  supplementary  report  to  this  effect 
will  soon  be  issued. 
Item  Analyses 

Another  way  to  assess  the  effectiveness  of  the  various  treatments  is  to 
consider  the  results  from  an  item  analysis  of  each  test.      Table    6   presents  sign- 
test  data  for  each  of  the  achievement  tests  for  each  of  the  pairs  of  groups: 
C  -  P,    A  -  C,    A  -  P.     Item  difficulty  levels   [(%  correct)  X  100]  were  computed 
by  item  for  each  group  and  direction  of  difference  noted. 


As  before,    the  significant  results  come  in  the  comparison  between  the    C 
and    P  groups.      The  table  shows  that  the    C   group  outperformed  the    P  group  by 
answering  correctly  a  significantly  larger  nvimber  of  items  on   6    of  the    9 
achievement  tests. 

These  results  seem  to  indicate  that  the  significant  differences  indicated  by 
the  _t   tests  were  not  functions  of  a  few  specific  items.      Rather,   we  strengthen 
our  conclusion  that  treatment  effects  are  being  manifested. 
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In  acklituMi,    wc   can   IdoR  at    i\cn\   types   [t  rur  -  lal  sc,     mulliph-   cFuMtf,    "supply"] 
and  tunctions  [roc. ill,    transfer,    application].     In  contrast  to  the   results  noted  in 
an  earlier    report   (Brown,     19^^),    this  yi'ar's   experimental   group  did  not  have 
superior  performance  on  any  particular  class  of  items;   the    C    group  did  better 
than  the    P    group  in  all  but  the  "true-false"   class.      The  only  other  significant 
difference  was  that  the    A   group  did  better  than  the    C    group  on  "  true-false" 
items;  this  result  seems  to  be  due  more  to  chance  than  to  treatment  effects. 

Another  result  noted  in  the  earlier  report  was  that  the  use  of  our  programed 
materials  led  to  homogeneous  performance  levels  on  the  program-related 
achievement  tests  among  students  of  varying  abilities.     Looking  at  the  distri- 
butions of  achievement  test  scores  by  means  of  Tables    2-4    certainly  does  not 
lead  to  such  a  conclusion  this  year.     In  fact,    total  variance  in  each  of  the   A-  C 
and   C  -  P   comparisons  is  more  often  larger  for  the  experimental  group  than  it 
is  for  the  control  group.     We  must  reject  a  "homogeneous  performance  levels" 

assertion  in  the  context  of  results  reported  in  this  study.     Notice,    however,    that 
j 

Our  "unit"    here  is  not  classes,    as  it  was  in  the  previous   study. 

An  additional  result  which  must  be  mentioned  came  more  from  the  preparation 

of  our  materials  than  from  their  trial.     This  "pedagogical"   result  is  a  collection 

;     of  new  devices  and  examples  and  presentation  schemes.     We  wished  to  provide  the 
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"pure"    students  with  a  rich  variety  of  experiences  within  the    programed  texts 
themselves  and  had  to  come  up  with  enough  approaches  to  a  topic  to  approximate 
a  trained  teacher's  presentation.     A  booklet  (UICSM  staff,    1963)  is  available 
which  describes  some  of  our  techniques.      It  is  interesting  to  note  that  some  of 
the  things  we  came  up  with  have  already  been  adopted  for  classroom  use  or  for 
use  in  new  "regular"  textbooks. 

DISCUSSION 

The  data  reported  in  this  paper  refer  to  ways  of  using  programed  materials 
in  the  classroom.     As  noted  earlier,    we  can  use  our  "pure"  mode  to  help 
establish  a  minimum  expectation  level  for  achievement  of  a  student  working  through 
the  materials  entirely  on  his   own.     It  is  a  minimum  level  in  that  we  foxind  that 
having  a  classroom  teacher  in  the  instructional  process  results  in  iraproved 
performance.      The  two  comparisons  we  have  reported  which  deal  with  the  "pure" 
mode  show  that  having  a  teacher  produces  significantly  better  results  than  the 
program  alone. 

The  establishment  of  a  minimiim  expectation  level  for  achievement  increases 
the  values  of  using  programed  materials  (a)  as  the  main  agent  of  instruction  for 
homebound  students  or     (b)   as  a  source  of  controlled  homew^ork  assignments,    or 


(i  )    Willi  iituciriits  who  should  study  matt- rial,     rftinclial  or  otht-rwiso,    different 
t  rom  that   of  the  majority  of  tlu-  class.      In  each  of  these  cases,    the  ability  to 
predict  a  mininiutii  level  of  achievement  from  independent  study  of  the  program 
would  help  a  teacher  make  best  use  of  the  programs  in  an  instructional  sequence. 

Prolonged  use  of  the  "pure"  mode  is  most  suited  to  research  and  we  had 
hoped  that  prolonged  use  of  the  "anticipating"  mode  would  be  most  suited  to  the 
classroom,    giving  a  very  efficient  procedure  for  use  of  our  programed  materials. 
Although  results  from  the  "anticipating"  mode  are  acceptable,    we  had  hoped 
they  would  be  even  better.     It  may  be  that  we  were  caught  by  a  "sameness"  factor; 
in  this  case  it  is  the  apparent  sameness  of  teaching  strategy.      The  word  'apparent' 
ise  used  because  the  perceived  difference  between  programed  and  teacher 
instruction  is  probably  far  greater  than  any  real  differences  we  could  build  into 
the  programed  texts  themselves.      And,    it  is  the  case  that,    for  every  new  topic, 
the  order  of  these  two  kinds  of  instruction  was  the   same,    first  the  program  and 
then  the  teacher. 

We  suspect  that  the  programed  texts  could  be  used  more  flexibly  than  was 
possible  under  the  prescriptions  of  the  "anticipating"  mode.  A  teacher  would 
have  to  bear  in  mind  the  introductional  emphasis  of  the  program,    but  could  well 
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plan  to  start  a  new  topic  with  classroom  discussion  late  in  a  class  period  and  let 
the  program  take  over  for  homework.      The  teacher  could  expect  a  certain  pro- 
ficiency from  students  who  had  completed  a  given  segment  of  a  program  and 
could  build  the  next  classroom  discussion  from  that  point.     Occasional  use  of 
this  order  of  teacher  and  program  would  clearly  curtail  the  sameness  effect 
noted  above.     In  addition,    it  seems  that  it  would  be  very  efficient. 

In  the  light  of  these  considerations,    it  may  be  the  case  that  programed 
materials  nnight  be  most  useful  when  packaged  as    topic -units    rather  than  as  a 
semester's  or  a  year's  worth  of  work.      Topic-units    would  be  very  manageable 
and  might  generate  very  positive  attitudes  due  to  their  short  length.     Using 
short  topic -units  might  also  enable  a  teacher  to  control  better  each  student's 
progress  than  is  possible  with  the  usual  programs,    thus  keeping  the  class 
together  for  maximum  benefit  from  classroom  discussions. 

The  fact  that  our  approach  to  programing  led  us  to  "invent"  new  examples 
and  presentation  schemes  warrants  the  suggestion  that  in  preparing  conventional 
texts  materials  one  could  in  tandem  prepare  programed  materials.      The  insights 
which  are  almost  forced  upon  one  in  creative  programing  of  a  topic  about  how  a 
student  learns  the  topic    piece -by-piece    should  be  very  valuable  in  writing  new 
texts,    or  even  in  making  revisions  of  an  established  text. 
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A    bv-pri)iUu  I    i>l  our  .ippio.n  h  to  p  i  <  .,■  r.i  m  i  n^ ,    as  was  mentioned  c-arlicr,    is 
that  we  can  use  the  proy  ratiied  texts  we   have  prepared  as  teache  r -t  raininj^ 
instruments.      A  commentary,    perhaps  proj^ramed,    to  the  student  program  could 
serve  to  point  out  the  various  pedagogical  techniques  used  in  the  progrann.      The 
commentary  could  also  make  suggestions  for  classroom  implementation  of  the 
new  devices  which  appear  in  the  program.      Even  a  teacher  without  mastery  of 
the  subject  matter  could  profit  from  such  training;  after  all,    the  obvious  function 
of  the  prograjned  texts  is  to  teach  the  subject  matter. 

SUMMARY 

The  data  from  a  matched-pair  analysis  of  achievement  test  results 
representing  a  control  group  and  groups  using  the  UICSM-prepared  programed 
materials  under  two  experimental  conditions  lead  to  the  following  conclusions: 

(1)  Programed  materials  prepared  by  experienced  UICSM  writers 
and  teachers  when  used  as  sole  agents  of  instruction  are  not  as 
effective  as  a  UICSM  teacher  trained  in  the  presentation  of  the 
same  content  material  and  using  a  regular  text. 

(2)  A  UICSM  teacher  using  the  programed  materials  to  introduce 
topics  to  a  class,    thus  anticipating  their  classroom  development, 
is  not  more  effective  than  a  comparable  teacher  independent  of 
the  programed  materials.     In  fact,    the  tw^o  procedures  lead  to 
roughly  equivalent  achievement  test  results. 

(3)  A  UICSM  teacher  using  the  programed  materials  to  introduce 
topics  to  a  class  is  more  effective  than  the  programed 
materials  used  as  sole  agents  of  instruction. 
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(4)      The  finding  from  other  studies  that  use  of  programed  materials 
results  in  more  homogeneous  achievement  over  varying  ability 
levels  is  not  supported  by  our  data  which  use  students,    not 
classes,    as  "units"   of  observation. 


We  note  that  the  data  reported  for  this  study  are  only  one  facet  of  the  total 
data  analysis  done.     On  the  basis  of  the  total  analysis,    a  revised  set  of  materials 
was  prepared  and  is  currently  undergoing  a  comprehensive  evaluation  which  will 
include  careful  analysis  of  time  data  as  well  as  and  in  conjunction  with  treatment 
results. 

In  the  actual  creative  programing  effort  needed  to  write  our  materials,    we 
found  important  gains  in  our  insights  into  how  a  student  learns  each  topic;  these 
insights  resulted  in  new  pedagogical  devices,    examples,    and  presentation 
schemes.      The  production  of  ideas  which  have  uses  apart  from  the  programed 
texts  is  a  dividend  that  enhances  the  worth  of  programing. 

Finally,    we  note  that  this   report  does  not  refer  to  specific  content  in  any  of 
the  comparisons,    the  emphasis  has  been  on  overall  effects  of  various  treatments. 
It  would  be  useful  to  consider  effects  across  specific  content  areas  which  recur 
through  the  Unit    1    sequence. 
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Appcjuhx    A 
Sunun.irv   ol   pa  rt  u  ipat  i  n^  t  la  b  sts  .     UICSM-IMP.     l9^>Z-6i 


Nuiiilx-  I-  ol  Numbf  r  ol 

St.  liool  classes  stucirnts  (loiidilioii  (jradc 

Talawanda  Hij^h  School  1  Z8  "   anlicipatinj^"  9 

Oxtord.    Ohio 

Pekin  Conimunity   High  School  2  6Z  "pure"  9 

Pekin,    Illinois 

St.    Louis   Preparatory  Seminary  1  34  "following"  9 

St.    Louis,    Missouri 

The   Principia,    Upper  School  1  18  "anticipating"  9 

St.    Louis,    Missouri 

Matignon  High  School  2  69  "pure"  9 

Cambridge,    Massachusetts 

St.    Rosalia  High  School  I  37  "following"  9 

Pittsburgh,    Pennsylvania 

Sacred  Heart  High  School  1  44  "anticipating"  9 

Pittsburgh,    Pennsylvania 

Marian  High  School  3  119  "pure"  9 

Framingham,    Massachusetts 

Sacred  Heart  High  School  1  42  control  9 

Newton  Centre,    Massachusetts 

Sacred  Heart  High  School  1  50  control  9 

Weymouth,    Massachusetts 

St.    Mary  High  School  1  43  control  9 

Brookline,    Massachusetts 

Immaculate  Conception  High  School  1  54  control  9 

Revere,    Massachusetts 

Cardinal  Spellman  High  School  1  40  control  9 

Brocton,    Massachusetts 

Vlt,    St.    Joseph  Acadenny  1  39  control  9 

^Brighton,    Massachusetts 

3t.    James  High  School  1  47  control  9 

fiaverhill,    Massachusetts 

Pontbonne  Academy  1  37  control  9 

Vlilton,    Massachusetts 
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Appendix    C 
Means  of  "Following"   Classes 


Test 


Class 


STEP  289.38  259.41 


TOGA  130.24 


95.  14 


1 

20.  14 

21.33 

2 

27.44 

17.19 

3 

23.82 

9.64 

4 

11.11 

6.70 

5 

12.  24 

3.53 

6 

28.28 

14.24 

7 

22.21 

13.  23 

8 

18.65 

13.23 

9 

8.92 

5.  23 

Unit  1 

21.21 

7.83 

N   =    37 


N   =    34 


AppiMidi  \     I) 

Valuos  of    I    for  differences  of  test  means  between 
each  two  samples    ri'prest'nting  a  given  tri'atmeril 
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Groupt 


Test 

P3-P1' 

STEP 

0.800 

TOGA 

0.389 

1 

-0.332 

2 

-0. 152 

3 

0.  123 

4 

1.712 

5 

0.829 

6 

1.635 

7 

1.883 

8 

1.004 

9 

1.625 

A, -A3 

Ci-Cp 

-1.686 

0.887 

-2.  105v/ 

1.948 

-0.828 

0.586 

-0.661 

0.252 

-1.203 

0.858 

-0.828 

0.840 

-0.592 

0.996 

-0.079 

-0.644 

-0.902 

0.546 

-0.934 

0.614 

-0.871 

0.539 

The  "pure"  members  of  the    A-P   matched  pairs  are    P^    and  the 
"pure"  members   of  the    C-P   matched  pairs  are    P^  ,    df    =    140. 


The  "anticipating"  members  of  the  A-  C  matched  pairs  are  A„ 
and  the  "anticipating"  members  of  the  A-P  matched  pairs  are 
A3;  df   =    103. 

The  "control"  members  of  the    C-P   matched  pairs  are    C,     and  the 
"control"  members  of  the    A-C   matched  pairs  are    C^  ;   df   =    141. 
v/  Significant  at     a    =    0.05. 
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